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Will covid-19 vaccines save lives? Current trials aren’t designed to tell 


US 


The world has bet the farm on vaccines as the solution to the pandemic, but the trials are not focused 
on answering the questions many might assume they are. Peter Doshi reports 


Peter Doshi associate editor 


As phase III trials of covid-19 vaccines reach their 
target enrolments, officials have been trying to project 
calm. The US coronavirus czar Anthony Fauci and 
the Food and Drug Administration leadership have 
offered public assurances that established procedures 
will be followed.’ 4 Only a “safe and effective” 
vaccine will be approved, they say, and nine vaccine 
manufacturers issued a rare joint statement pledging 
not to prematurely seek regulatory review.° 


But what will it mean exactly when a vaccine is 
declared “effective”? To the public this seems fairly 
obvious. “The primary goal of a covid-19 vaccine is 
to keep people from getting very sick and dying,” a 
National Public Radio broadcast said bluntly.° 
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Peter Hotez, dean of the National School of Tropical 
Medicine at Baylor College of Medicine in Houston, 
said, “Ideally, you want an antiviral vaccine to do 
two things... first, reduce the likelihood you will get 
severely ill and go to the hospital, and two, prevent 
infection and therefore interrupt disease 
transmission.” 


Yet the current phase III trials are not actually set up 
to prove either (table 1). None of the trials currently 
under way are designed to detect a reduction in any 
serious outcome such as hospital admissions, use of 
intensive care, or deaths. Nor are the vaccines being 
studied to determine whether they can interrupt 
transmission of the virus. 
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Table 1 | Characteristics of ongoing phase III covid-19 vaccine trials 
































Moderna Pfizer AstraZeneca (US) AstraZeneca (UK) Janssen Sinopharm* Sinovac 
Vaccine name mRNA-1273 BNT162 AZD1222 AZD1222 Ad26.COV2.S Sinopharm vaccine Sinovac CoronaVac 
Registration No NCT04470427 NCT04368728 NCTO4516746 NCT04400838 (UK), NCT04505722 NCTO4510207 NCT04456595 

NCT04536051 
(Brazil), 
NCT04444674 (South 
Africa) 

Target enrolment 30 000 43 998 30 000 19 330 60 000 45 000 8870 
Ages eligible 18+ 12+ 18+ 5-12, 18+ 18+ 18+ 18+ 
Protocol publicly Y V v Nt Vi N 
available 
Notable excluded populations: 
Children and Excluded Many excluded Excluded 13-17 excluded Excluded Excluded Excluded 
adolescents 
Immunocompromised Excluded Excluded Excluded Excluded Excluded Excluded Excluded 
patients 
Pregnant or Excluded Excluded Excluded Excluded Excluded Excluded Excluded 
breastfeeding women 
Endpoints undergoing formal study#: 
Prevention of Yi Mf Ve Vf v Presumably§ Y 
symptomatic disease 
in vaccine recipient 
Reduction in severe N N N NG N N N 
covid-19 (hospital 
admission, ICU, or 
death) 
Interruption of N N N N N N N 


transmission (person 
to person spread) 





* This trial is separately randomising an inactivated SARS-CoV-2 vaccine (Vero cell) manufactured by Wuhan Institute of Biological Products Co and Beijing Institute of Biological Products Co. 





t AstraZeneca has released the protocol for its stalled US trial but not its trial in UK, Brazil, and South Africa. 


+ Endpoints “undergoing formal study” include those listed as primary outcomes in ClinicalTrials.gov, publicly available study protocols, or those not listed as primary outcomes, but the company has confirmed that the 


study is powered sufficiently to find an effect (if one exists). 


§ Sinopharm lists “incidence of COVID-19 cases” as a primary efficacy endpoint in its ClinicalTrials.gov entry. 


(Trial registration (NCT04444674) lists the following primary endpoint: “Determine if there is a reduction of severe and non-severe COVID-19 disease in HIV-negative adults.” This suggests a composite outcome that 


includes non-severe disease. 


Evaluating mild, not severe, disease 


In a September interview Medscape editor in chief Eric Topol 
pondered what counts as a recorded “event” in the vaccine trials. 
“We're not talking about just a PCR [polymerase chain reaction 
test]-positive mild infection. It has to be moderate to severe illness 
to qualify as an event, correct?” he asked.® 


“That’s right,” concurred his guest, Paul Offit, a vaccinologist who 
sits on the FDA advisory committee that may ultimately recommend 
the vaccines for licence or emergency use authorisation. 


But that’s not right. In all the ongoing phase III trials for which 
details have been released, laboratory confirmed infections even 
with only mild symptoms qualify as meeting the primary endpoint 
definition.° ” In Pfizer and Moderna’s trials, for example, people 
with only a cough and positive laboratory test would bring those 
trials one event closer to their completion. (If AstraZeneca’s ongoing 
UK trial is designed similarly to its “paused” US trial for which the 
company has released details, a cough and fever with positive PCR 
test would suffice.) 


Part of the reason may be numbers. Severe illness requiring hospital 
admission, which happens in only a small fraction of symptomatic 
covid-19 cases, would be unlikely to occur in significant numbers 


in trials. Data published by the US Centers for Disease Control and 
Prevention in late April reported a symptomatic case hospitalisation 
ratio of 3.4% overall, varying from 1.7% in 0-49 year olds and 4.5% 
in 50-64 year olds to 7.4% in those 65 and over.'? Because most 
people with symptomatic covid-19 experience only mild symptoms, “ 
even trials involving 30 000 or more patients would turn up 
relatively few cases of severe disease. 


In the trials, final efficacy analyses are planned after just 150 to 160 
“events,”—that is, a positive indication of symptomatic covid-19, 
regardless of severity of the illness. 


Yet until vaccine manufacturers began to release their study 
protocols in mid-September, trial registries and other publicly 
released information did little to dispel the notion that it was severe 
covid-19 that the trials were assessing. Moderna, for example, called 
hospital admissions a “key secondary endpoint” in statements to 
the media. And a press release from the US National Institutes of 
Health reinforced this impression, stating that Moderna’s trial “aims 
to study whether the vaccine can prevent severe covid-19” and 
“seeks to answer if the vaccine can prevent death caused by 
covid-19.”2° 


But Tal Zaks, chief medical officer at Moderna, told The BMJ that 
the company’s trial lacks adequate statistical power to assess those 
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outcomes. “The trial is precluded from judging [hospital 
admissions], based on what is a reasonable size and duration to 
serve the public good here,” he said. 


Hospital admissions and deaths from covid-19 are simply too 
uncommon in the population being studied for an effective vaccine 
to demonstrate statistically significant differences in a trial of 30 
000 people. The same is true of its ability to save lives or prevent 
transmission: the trials are not designed to find out. 


Zaks said, “Would I like to know that this prevents mortality? Sure, 
because I believe it does. I just don’t think it’s feasible within the 
timeframe [of the trial]|—too many would die waiting for the results 
before we ever knew that.” 


Stopping transmission 


What about Hotez’s second criterion, interrupting virus 
transmission, which some experts have argued”’ should be the most 
important test in phase III studies? 


“Our trial will not demonstrate prevention of transmission,” Zaks 
said, “because in order to do that you have to swab people twice a 
week for very long periods, and that becomes operationally 
untenable.” 


He repeatedly emphasised these “operational realities” of running 
a vaccine trial. “Every trial design, especially phase III, is always a 
balancing act between different needs,” he said. “If you wanted to 
have an answer on an endpoint that happens at a frequency of one 
1oth or one fifth the frequency of the primary endpoint, you would 
need a trial that is either 5 or 10 times larger or you’d need a trial 
that is 5 or 10 times longer to collect those events. Neither of these, 
I think, are acceptable in the current public need for knowing 
expeditiously that a vaccine works.” 


Zaks added, “A 30 000 [participant] trial is already a fairly large 
trial. If you’re asking for a 300 000 trial then you need to talk to the 
people who ate paying for it, because now you’re talking about not 
a $500m to $1bn trial, you’re talking about something 10 times the 
size. And I think the public purse and operational capabilities and 
capacities we have are rightly spent not betting the farm on one 
vaccine but, as Operation Warp Speed [the US government’s covid-19 
vaccine plan] is trying to do, making sure that we’re funding several 
vaccines in parallel.” 


Debating endpoints 


Still, it’s fair to say that most of the general public assumes that the 
whole point of the current trials, besides testing safety (box 1), is to 
see whether the vaccine can prevent bad outcomes. “How do you 
reconcile that?” The BMJ asked Zaks. 


Box 1: Safety and side effects 





History shows many examples of serious adverse events from vaccines 
brought to market in periods of enormous pressure and expectation. 
There were contaminated polio vaccines in 1955, cases of Guillain-Barré 
syndrome in recipients of flu vaccines in 1976, and narcolepsy linked to 
one brand of influenza vaccine in 2009.1° 19 

“Finding severe rare adverse events will require the study of tens of 
thousands of patients, but this requirement will not be met by early 
adoption of a product that has not completed its full trial evaluation,” 
Harvard drug policy researchers Jerry Avorn and Aaron Kesselheim recently 
wrote in JAMA.2° 

Covid-19 vaccine trials are currently designed to tabulate final efficacy 
results once 150 to 160 trial participants develop symptomatic 
covid-19—and most trials have specified at least one interim analysis 
allowing for the trials to end with even fewer data accrued. 
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Medscape’s Eric Topol has been a vocal critic of the trials’ many interim 
analyses. “These numbers seem totally out of line with what would be 
considered stopping rules,” he says. “I mean, you’re talking about giving 
a vaccine with any of these programmes to tens of millions of people. 
And you’re going to base that on 100 events?”® 

Great uncertainty remains over how long a randomised trial of a vaccine 
will be allowed to proceed. If efficacy is declared, one possibility is that 
the thousands of volunteers who received a saline placebo would be 
offered the active vaccine, in effect ending the period of randomised 
follow-up. Such a move would have far reaching implications for our 
understanding of vaccines’ benefits and harms, rendering uncertain our 
knowledge of whether the vaccines can reduce the risk of serious covid-19 
disease and precluding any further ability to compare adverse events in 
the experimental versus the placebo arm. 

“It'll be a decision we'll have to take at that time. We have not committed 
one way or another,” Moderna’s Tal Zaks told The BMJ. “It will be a 
decision where FDA and NIH will also weigh in. And it will be probably a 
very difficult decision, because you will be weighing the benefit to the 
public in continuing to understand the longer term safety by keeping 
people on placebo and the expectation of the people who have received 
placebo to be crossed over now that it has been proved effective.” 


“Very simply,” he replied. “Number one, we have a bad outcome 
as our endpoint. It’s covid-19 disease.” Moderna, like Pfizer and 
Janssen, has designed its study to detect a relative risk reduction 
of at least 30% in participants developing laboratory confirmed 
covid-19, consistent with FDA and international guidance.”! *? 


Number two, Zaks pointed to influenza vaccines, saying they protect 
against severe disease better than mild disease. To Moderna, it’s 
the same for covid-19: if its vaccine is shown to reduce symptomatic 
covid-19, it will be confident it also protects against serious 
outcomes. 


But the truth is that the science remains far from clear cut, even for 
influenza vaccines that have been used for decades. Although 
randomised trials have shown an effect in reducing the risk of 
symptomatic influenza, such trials have never been conducted in 
elderly people living in the community to see whether they save 
lives. 


Only two placebo controlled trials in this population have ever been 
conducted, and neither was designed to detect any difference in 
hospital admissions or deaths.73 Moreover, dramatic increases in 
use of influenza vaccines has not been associated with a decline in 
mortality (box 2).? 


Box 2: Not enrolling enough elderly people or minorities 





A vaccine that has been proved to reduce the risk of symptomatic disease 
by a certain proportion should, you might think, reduce serious outcomes 
such as hospital admissions and deaths in equal proportion. 

Peter Marks, an FDA official with responsibility over vaccine approvals, 
recently stated as much about influenza vaccination, which “only prevents 
flu in about half the people who get it. And yet that’s very important 
because that means that it leads to half as many deaths related to 


influenza each year.”74 


But when vaccines are not equally effective in all populations the theory 
breaks down. 


If frail elderly people, who are understood to die in disproportionate 


numbers from both influenza2> and covid-19, are not enrolled into vaccine 
trials in sufficient numbers to determine whether case numbers are 
reduced in this group, there can be little basis for assuming any benefit 
in terms of hospital admissions or mortality. Whatever reduction in cases 
is seen in the overall study population (most of which may be among 
healthy adults), this benefit may not apply to the frail elderly 
subpopulation, and few lives may be saved. 
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This is hard to evaluate in the current trials because there are large gaps 
in the types of people being enrolled in the phase Ill trials (table 1). 
Despite recruiting tens of thousands, only two trials are enrolling children 
less than 18 years old. All exclude immunocompromised people and 
pregnant or breastfeeding women, and though the trials are enrolling 
elderly people, few or perhaps none of the studies would seem to be 
designed to conclusively answer whether there is a benefit in this 
population, despite their obvious vulnerability to covid-19. 

“Adults over 65 will be an important subgroup that we will be looking 
at,” Moderna’s Zaks told The BMJ. “That said . . . any given study is 
powered for its primary endpoint—in our case covid-19 disease 
irrespective of age.” 

Al Sommer, dean emeritus of the Johns Hopkins School of Public Health, 
told The BM), “|f they have not powered for evidence of benefit in the 
elderly, | would find that a significant, unfortunate shortcoming.” He 
emphasised the need for “innovative follow-up studies that will enable 
us to better determine the direct level of protection immunisation has 
on the young and, separately, the elderly, in addition to those at the 
highest risk of severe disease and hospitalisation.” 


One view is that trial data should be there for all target populations. “If 
we don’t have adequate data in the greater than 65 year old group, then 
the greater than 65 year old person shouldn’t get this vaccine, which 
would be a shame because they’re the ones who are most likely to die 


from this infection,” said vaccinologist Paul Offit.° “We have to generate 
those data,” he said. “I can’t see how anybody—the Data and Safety 
Monitoring Board or the FDA Vaccine Advisory Committee, or FDA 
decision-makers—would ever allow a vaccine to be recommended for 
that group without having adequate data.” 

“| feel the same way about minorities,” Offit added. “You can’t convince 
minority populations to get this vaccine unless they are represented in 
these trials. Otherwise, they’re going to feel like they’re guinea pigs, and 
understandably so.” 
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